Wearable device and computer enabled feedback for user task assistance

ABSTRACT

Detecting spatial positioning between a user&#39;s hands and an object, using a wearable device, to provide feedback to a user. Motion data of a person&#39;s body motion while performing an action can be received at a computer. The motion data can include sensor data from sensors at a location where the sensors detect the person&#39;s body motion. The sensors can include a wearable device on the person&#39;s body. The computer can be used to model the person&#39;s body motion using the motion data. A set of parameters for acceptable motions is determined based on an action risk assessment of the action. Feedback can be initiated to the person, using the wearable device, based on the person&#39;s body motion exceeding a body motion threshold based on the set of parameters for the acceptable motions and the model.

BACKGROUND

The present disclosure relates to generating feedback to a personperforming a physical task, using a wearable device and a computer, fortask assistance or information feedback for the user performing thetask.

In one example, a person performing a physical task can be in danger ofinjuring themselves in many ways. For instance, a person performing acutting or scraping task in a kitchen with utensils can be at risk forinjuring themselves with sharp utensils, e.g., a knife. In one example,a person or user can use a knife for cutting, and be unaware of thecloseness of one hand to another, or their body motions puttingthemselves at risk, such as arm or hand motions with a sharp instrument,which can result in an injury.

SUMMARY

The present disclosure recognizes the shortcomings and problemsassociated with current techniques for using artificial intelligence(AI) enabled feedback for task assistance for a user, and morespecifically, detecting spatial positioning between a user's hands andan object, using a wearable device, to provide feedback to a user.

In an aspect according to the present invention, a computer-implementedmethod for detecting spatial positioning between a user's hands and anobject, using a wearable device, to provide feedback to a user includesreceiving, at a computer, motion data of a person's body motion whileperforming an action. The motion data including sensor data from sensorsat a location, and the sensors detecting the person's body motion, andthe sensors including a wearable device on the person's body. The methodfurther including modeling, using the computer, the person's body motionusing the motion data, and determining, using the computer, a set ofparameters for acceptable motions based on an action risk assessment ofthe action. The method includes initiating feedback, using the wearabledevice, to the person based on the person's body motion exceeding a bodymotion threshold based on the set of parameters for the acceptablemotions and the model.

In a related aspect, the feedback can include haptic feedback generatedusing the wearable device.

In a related aspect, the method further includes determining the bodymotion threshold by comparing the set of parameters for the acceptablemotion to the model of the person's body motions.

In a related aspect, the method further includes determining the bodymotion threshold by comparing the set of parameters for the acceptablemotion to the model of the person's body motions, and the comparingincludes a risk assessment of the body motions based on the riskassessment of the action.

In a related aspect, the method further includes determining the bodymotion threshold by comparing the set of parameters for the acceptablemotion to the model of the person's body motions, wherein the comparingincludes a risk assessment of the body motions based on the action riskassessment to determine a risk factor for a particular motion.

In a related aspect, the method further including determining the bodymotion threshold by comparing the set of parameters for the acceptablemotion to the model of the person's body motions, wherein the comparingincludes a risk assessment of the body motions based on the action riskassessment to determine a risk factor for a particular motion, whereinthe body motion threshold is based on the risk factor for the particularmotion.

In a related aspect, the method further including determining the bodymotion threshold by comparing the set of parameters for the acceptablemotion to the model of the person's body motions, wherein the comparingincludes a risk assessment of the body motions based on the action riskassessment to determine a risk factor for a particular motion, whereinthe body motion threshold is based on the risk factor for the particularmotion, wherein the body motion threshold does not exceed a risk factorvariable.

In a related aspect, the risk factor variable is based on the set ofparameters for acceptable motions.

In a related aspect, the initiating of the feedback includes sending acommunication to the wearable device.

In a related aspect, the body motion includes hand motions of theperson.

In a related aspect, the method further including sending acommunication to the wearable device enabling warnings from the wearabledevice when the user's hand motions exceed a threshold based on the setof the parameters for the acceptable motions.

In a related aspect, the sensors at the location including a combinationof one or more site sensors and a wearable sensor.

In a related aspect, the site sensors include a camera, an audio sensoror microphone device, and the wearable sensor includes an accelerometer.

In a related aspect, the wearable sensor is a wrist worn smart device orsmartwatch.

In a related aspect, the modeling of the person's body motion, includingmodeling a user's hand motions using the motion data.

In a related aspect, the modeling of the person's body motion includesusing the motion data and a spring-mass-damper system data.

In a related aspect, a communication to the wearable device enablingwarnings from the wearable device when the user's hand motions exceedthe set of the parameters for the acceptable motions.

In a related aspect, the method further including updating the model ofthe person's body motion with updated motion data.

In another aspect, according to the present invention, a system fordetecting spatial positioning between a user's hands and an object,using a wearable device, provides feedback to a user, which includes: acomputer system comprising; a computer processor, a computer-readablestorage medium, and program instructions stored on the computer-readablestorage medium being executable by the processor, to cause the computersystem to perform the following functions to; receive, at a computer,motion data of a person's body motion while performing an action, themotion data including sensor data from sensors at a location, thesensors detecting the person's body motion, and the sensors including awearable device on the person's body; model, using the computer, theperson's body motion using the motion data; determine, using thecomputer, a set of parameters for acceptable motions based on an actionrisk assessment of the action; and initiate feedback, using the wearabledevice, to the person based on the person's body motion exceeding a bodymotion threshold based on the set of parameters for the acceptablemotions and the model.

In another aspect, according to the present invention, a computerprogram product for detecting spatial positioning between a user's handsand an object, using a wearable device, provides feedback to a user. Thecomputer program product comprising a computer readable storage mediumhaving program instructions embodied therewith, the program instructionsexecutable by a computer to cause the computer to perform functions, bythe computer, comprising the functions to; receive, at a computer,motion data of a person's body motion while performing an action, themotion data including sensor data from sensors at a location, thesensors detecting the person's body motion, and the sensors including awearable device on the person's body; model, using the computer, theperson's body motion using the motion data; determine, using thecomputer, a set of parameters for acceptable motions based on an actionrisk assessment of the action; and initiate feedback, using the wearabledevice, to the person based on the person's body motion exceeding a bodymotion threshold based on the set of parameters for the acceptablemotions and the model.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

These and other objects, features and advantages of the presentinvention will become apparent from the following detailed descriptionof illustrative embodiments thereof, which is to be read in connectionwith the accompanying drawings. The various features of the drawings arenot to scale as the illustrations are for clarity in facilitating oneskilled in the art in understanding the invention in conjunction withthe detailed description. The drawings are discussed forthwith below.

FIG. 1 is a schematic block diagram illustrating an overview of asystem, system features or components, and methodology for detectingspatial positioning between a user's hands and an object, using awearable device, to provide feedback to a user, according to anembodiment of the present disclosure.

FIG. 2 is a flow chart illustrating a method, implemented using thesystem shown in FIG. 1 , for detecting spatial positioning between auser's hands and an object, using a wearable device, to provide feedbackto a user, which can include an artificial intelligence (AI) model anddata analysis, according to an embodiment of the present disclosure.

FIG. 3 is a functional schematic block diagram showing a series ofoperations and functional methodologies, for instructional purposesillustrating functional features of the present disclosure associatedwith the embodiments shown in the FIGS., which can be implemented, atleast in part, in coordination with the system shown in FIG. 1 , fordetecting spatial positioning between a user's hands and an object,using a wearable device, to provide feedback to a user.

FIG. 4A is a flow chart illustrating another method, which continuesfrom the flow chart of FIG. 2 , for detecting spatial positioningbetween a user's hands and an object, using a wearable device, toprovide feedback to a user.

FIG. 4B is a flow chart illustrating another method, which continuesfrom the flow chart of FIG. 2 , for detecting spatial positioningbetween a user's hands and an object, using a wearable device, toprovide feedback to a user.

FIG. 5 is a flow chart illustrating another method according to anembodiment of the present invention, for detecting spatial positioningbetween a user's hands and an object, using a wearable device, toprovide feedback to a user.

FIG. 6A is a block diagram illustrating another system according to anembodiment of the present invention, for detecting spatial positioningbetween a user's hands and an object, using a wearable device, toprovide feedback to a user, and a graph depicting motions along an axis.

FIG. 6B is a graph illustrating a notification waveform related to thesystem of FIG. 6A.

FIG. 7A is a block diagram illustrating another system according to anembodiment of the present invention, for detecting spatial positioningbetween a user's hands and an object, using a wearable device, toprovide feedback to a user, and showing a visual indicator.

FIG. 7B is a graph illustrating a notification waveform related to thesystem of FIG. 7A.

FIG. 8A is a block diagram illustrating another system according to anembodiment of the present invention, for detecting spatial positioningbetween a user's hands and an object, using a wearable device, toprovide feedback to a user, and includes a user's hands moving closer toone another.

FIG. 8B is a graph illustrating an notification waveform related to thesystem of FIG. 8A.

FIG. 9 is a block diagram illustrating another system according to anembodiment of the present invention, for detecting spatial positioningbetween a user's hands and an object, using a wearable device, toprovide feedback to a user, and includes a visual indicator and adamping force estimation.

FIG. 10 is a schematic block diagram depicting a computer systemaccording to an embodiment of the disclosure which may be incorporated,all or in part, in one or more computers or devices shown in FIG. 1 ,and cooperates with the systems and methods shown in the FIGS.

FIG. 11 is a schematic block diagram of a system depicting systemcomponents interconnected using a bus. The components for use, in all orin part, with the embodiments of the present disclosure, in accordancewith one or more embodiments of the present disclosure.

FIG. 12 is a block diagram depicting a cloud computing environmentaccording to an embodiment of the present invention.

FIG. 13 is a block diagram depicting abstraction model layers accordingto an embodiment of the present invention.

DETAILED DESCRIPTION

The following description with reference to the accompanying drawings isprovided to assist in a comprehensive understanding of exemplaryembodiments of the invention as defined by the claims and theirequivalents. The description includes various specific details to assistin that understanding, but these are to be regarded as merely exemplary,and assist in providing clarity and conciseness. Accordingly, those ofordinary skill in the art will recognize that various changes andmodifications of the embodiments described herein can be made withoutdeparting from the scope and spirit of the invention. In addition,descriptions of well-known functions and constructions may be omitted.

The terms and words used in the following description and claims are notlimited to the bibliographical meanings, but are merely used to enable aclear and consistent understanding of the invention. Accordingly, itshould be apparent to those skilled in the art that the followingdescription of exemplary embodiments of the present invention isprovided for illustration purpose only and not for the purpose oflimiting the invention as defined by the appended claims and theirequivalents.

It is to be understood that the singular forms “a,” “an,” and “the”include plural referents unless the context clearly dictates otherwise.Thus, for example, reference to “a component surface” includes referenceto one or more of such surfaces unless the context clearly dictatesotherwise.

Embodiments and Examples

Embodiments and figures of the present disclosure may have the same orsimilar components as other embodiments. Such figures and descriptionsof illustrate and explain further examples and embodiments according tothe present disclosure.

Referring to FIGS. 1, 2 and 3 , according to embodiments of the presentdisclosure, a computer-implemented method 200 for detecting spatialpositioning between a user's hands and an object, using a wearabledevice, to provide feedback to a user includes features described below.Embodiments of the present disclosure include operational actions and/orprocedures. The computer-implemented method 200 includes a series ofoperational blocks for implementing an embodiment according to thepresent disclosure which can include the system shown in FIG. 1 . Theoperational blocks of the methods and systems according to the presentdisclosure can include techniques, mechanism, modules, and the like forimplementing the functions of the operations in accordance with thepresent disclosure.

Referring to FIGS. 1, 2 and 3 , the method 200 includes receiving, at acomputer 131, motion data 308 of a person's or user's 120 body motion302 while performing an action 304, as in block 204. Body motion caninclude one or more hand motions, for instance, while a person or useris using their hands to complete a task. A task can include for example,using a tool in any number of work/labor intensive operations, orcutting tool in a kitchen. Embodiments of the present disclosure canassist, for example, using all or in part artificial intelligence (AI) aperson needing assistance or additional confirmation or safety whenperforming a task. Additionally, a task can include any task requiringbody movement including walking or hand motions for any task includingall daily tasks a person may need to perform. The motion data 308includes sensor data 312 from one or more sensors 144 at a location 140,the sensors detecting the person's body motion which can be detected bythe sensors and generate the motion data 308. The sensors 144 includinga wearable device 148 on the person's 120 body, as also in block 204.

The method 200 further includes modeling, using the computer, theperson's body motion using the motion data thereby generating a model320, as in block 208. The model can be generated using a learning engineor modeling module 192 of a computer system 190 which can be all or inpart of an Artificial Intelligence (AI) system which communicates withthe computer 131 and/or a control system 170. Such a computer system 190can include or communicate with a knowledge corpus or historicaldatabase 196. A model can also be generated by an AI system such as anoutput at least in part of an AI system analysis using machine learning.

When the model is acceptable at block 210, the method can proceed toblock 212. In one example, an acceptable model can include a modelmeeting specified parameters. In another example, an acceptable modelcan be a model which has undergone several iterations. When the model isnot acceptable, the method can return to block 208.

The method 200 includes determining, using the computer, a set ofparameters 324 for acceptable motions 326 based on an action riskassessment 328 of the action 304, as in block 212.

When the set of parameters are acceptable, at block 214, the method canproceed to block 216. In one example, an acceptable set of parameterscan include parameters meeting specified criteria. When the parametersare not acceptable, the method can return to block 212.

The method 200 includes initiating feedback 330, using the wearabledevice 148, to the person 120 based on the person's body motion 302exceeding a body motion threshold 334 based on the set of parameters 324for the acceptable motions 326 and the model 320, as in block 216.

In one example, the feedback can include haptic feedback generated usingthe wearable device.

In another example, the method can include determining the body motionthreshold by comparing the set of parameters for the acceptable motionto the model of the person's body motions.

In another example, the method can include determining the body motionthreshold by comparing the set of parameters for the acceptable motionto the model of the person's body motions. The comparing includes a riskassessment of the body motions based on the risk assessment of theaction. The action can include a person bumping into a wall, or a personmoving close to a wall or a step, or in another example, a person usinga sharp object coming close to another hand.

In another example, the method includes determining the body motionthreshold by comparing the set of parameters for the acceptable motionto the model of the person's body motions. The comparing includes a riskassessment of the body motions based on the action risk assessment todetermine a risk factor for a particular motion. A risk factor caninclude a person approaching a step, or a person using a sharp objectcoming close to the other hand.

In another example, the method further including determining the bodymotion threshold by comparing the set of parameters for the acceptablemotion to the model of the person's body motions. The comparing includesa risk assessment of the body motions based on the action riskassessment to determine a risk factor for a particular motion, and thebody motion threshold can be based on the risk factor for the particularmotion.

Referring to FIG. 4A, another method 400 according to the presentdisclosure can continue from block 204 from the method 200 shown in FIG.2 . The method 400 can include determining the body motion threshold bycomparing the set of parameters for the acceptable motion to the modelof the person's body motions, as in block 404. The comparing can includedetermining a risk assessment of the body motions based on the actionrisk assessment to determine a risk factor for a particular motion, asin block 406. The method can include basing the body motion threshold onthe risk factor for the particular motion, as in block 408. The methodincludes determining when the body motion threshold does not exceed arisk factor variable, as in block 410. The method 400 continues to block208 of the method 200.

The method 400 can further include the risk factor variable being basedon the set of parameters for acceptable motions.

In one example the initiating of the feedback can include sending acommunication to the wearable device.

In another example, the body motion includes hand motions of the person.

In another example, the method further includes sending a communicationto the wearable device enabling warnings from the wearable device whenthe user's hand motions exceed a threshold based on the set of theparameters for the acceptable motions.

In another example, the sensors at the location include a combination ofone or more site sensors and a wearable sensor.

In another example, the site sensors include a camera, an audio sensoror microphone device, and the wearable sensor includes an accelerometer.

In another example, the wearable sensor is a wrist worn smart device orsmartwatch.

In another example, the modeling of the person's body motion, includingmodeling a user's hand motions using the motion data.

In another example, the modeling of the person's body motion includesusing the motion data and a spring-mass-damper system data.

In another example, a communication to the wearable device enableswarnings from the wearable device when the user's hand motions exceedthe set of the parameters for the acceptable motions.

Referring to FIG. 4B, a method 450 can continue from block 216 of themethod 200 shown in FIG. 2 . The method 450 include updating the modelof the person's body motion with updated motion data, as in block 454.The method 450 includes updating the model of the person's body motion,as in block 456. The method includes continuing the block 208 of themethod 200.

The method 450 can further include iteratively generating the model toproduce updated models.

The computer 131 can be integral to or communicating with a device 148.A computer 190 remote from the device 148 can electronicallycommunicate, in all or in part, with the computer 172 as part of thecontrol system 170. The control system can include the computer 172having a computer readable storage medium 173 which can store one ormore programs 174, and a processor 175 for executing programinstructions. The control system can also include a storage medium whichcan include registration and/or account data 182 and profiles 183 ofusers or entities (such entities can include robotic entities) as partof user accounts 181. User accounts 181 can be stored on a storagemedium 180 which is part of the control system 170. The user accounts181 can include registrations and account data 182 and user profiles183. The control system can also include a computer 172 having acomputer readable storage medium 173 which can store programs or codeembedded on the storage medium. The program code can be executed by aprocessor 175. The computer 172 can communicate with a database 176. Thecontrol system 170 can also include a database 176 for storing all orpart of such data as described above, and other data.

The control system can also communicate with a computer system 190 whichcan include a learning engine/module 192 and a knowledge corpus ordatabase 196. The computer system 190 can also communicate with thecomputer 131 of the device 130 and can be remote from the user device130. In another example, the computer system 190 can be all or part ofthe control system, or all or part of the device 130. The depiction ofthe computer system 190 as well as the other components of the system100 are shown as one example according to the present disclosure.

The new or different AI (Artificial Intelligence) ecosystem, ortechnology/communication or IT (Information Technology) ecosystem caninclude a local communications network 152 which can communicate withthe communications network 160. The system 100 can include a learningengine/module 192, which can be at least part of the control system orcommunicating with the control system, for generating a model orlearning model. In one example, the learning model can model workflow ina new AI or IT ecosystem for machine/devices in the new ecosystem.

In another example, the computer 131 can be part of a device 130. Thecomputer can include a processor 132 and a computer readable storagemedium 134 where an application 135 can be stored which can in oneexample, embody all or part of the method of the present disclosure. Theapplication can include all or part of instructions to implement themethod of the present disclosure, embodied in code and stored on acomputer readable storage medium. The device 148 can include a display.The device 148 can operate, in all or in part, in conjunction with aremote server by way of a communications network 160, for example, theInternet.

Other Embodiments and Examples

Referring to FIG. 1 , the device 148, also can be referred to as a userdevice or an administrator's device, includes a computer 131 having aprocessor 132 and a storage medium 134 where an application 135, can bestored. The application can embody the features of the method of thepresent disclosure as instructions. The user can connect to a learningengine 150 using the device 130. The device 130 which includes thecomputer 131 and a display or monitor 138. The application 135 canembody the method of the present disclosure and can be stored on thecomputer readable storage medium 134. The device 130 can further includethe processor 132 for executing the application/software 135. The device130 can communicate with a communications network 160, e.g., theInternet.

It is understood that the user device 130 is representative of similardevices which can be for other users, as representative of such devices,which can include, mobile devices, smart devices, laptop computers etc.

In one example, the system of the present disclosure can include acontrol system 170 communicating with the user device 130 via acommunications network 160. The control system can incorporate all orpart of an application or software for implementing the method of thepresent disclosure. The control system can include a computer readablestorage medium 180 where account data and/or registration data 182 canbe stored. User profiles 183 can be part of the account data and storedon the storage medium 180. The control system can include a computer 172having computer readable storage medium 173 and software programs 174stored therein. A processor 175 can be used to execute or implement theinstructions of the software program. The control system can alsoinclude a database 176.

In another example and embodiment, profiles can be saved for entitiessuch as users, participants, operators, human operators, or roboticdevices. Such profiles can supply data regarding the user and history ofdeliveries for analysis. In one example, a user can register or createan account using the control system 170 which can include one or moreprofiles 183 as part of registration and/or account data 182. Theregistration can include profiles for each user having personalizeddata. For example, users can register using a website via their computerand GUI (Graphical User Interface) interface. The registration oraccount data 182 can include profiles 183 for an account 181 for eachuser. Such accounts can be stored on the control system 170, which canalso use the database 176 for data storage. A user and a related accountcan refer to, for example, a person, or an entity, or a corporateentity, or a corporate department, or another machine such as an entityfor automation such as a system using, in all or in part, artificialintelligence.

Additionally, the method and system is discussed with reference to FIG.3 , which is a functional system 300 which includes components andoperations for embodiments according to the present disclosure, and isused herein for reference when describing the operational steps of themethods and systems of the present disclosure. Additionally, thefunctional system 300, according to an embodiment of the presentdisclosure, depicts functional operations indicative of the embodimentsdiscussed herein.

Referring to FIG. 3 , in one embodiment according to the presentdisclosure, a system 300 can be used to identify objects related to anevent for use regarding the event by using networked computer systemresources. In FIG. 3 similar components may have the same referencenumerals as the system 100 shown in FIG. 1 , the system 300 can includeor operate in concert with a computer implemented method as shown inFIGS. 1 and 2 .

More Embodiments and Examples

In one example, one or more objects held by a user, such as sharpcutting instruments for instance a knife, can pose a risk of injury to aperson whether in a workplace, home, or kitchen, or using sharpinstruments as part of a job function.

An embodiment of the present disclosure can adapt audio-based safetynotification playout frequency and amplitude with respect to relativehand movement, and related to a vertical cut of tool and movement/cutspeed. A system can take sensor data (for example, an image from acamera, accelerometer, audio sample) from a wrist strapped smart device,with a sensitivity and safety threshold as input and generate audiotones at a higher frequency and amplitude if hands (free hand and toolhand) are closing-in. A lower frequency and amplitude can occur whenhands are moving-away. A higher frequency and amplitude can occur iftool hand is cutting-in. Lower frequency and amplitude can occur if atool hand is moving-up the cut.

Referring to FIG. 5 , in one embodiment according to the presentdisclosure, a method 500 includes receiving an input of audio samples asin block 504. A computer is used to determine audio-based hand movementestimations, as in block 508. In one example using a cutting tool,(e.g., a knife), the determination and calculations can include adistance moved by a tool hand (e.g., the hand holding the tool), and atool hand movement velocity, and a cut depth.

The computer is used for estimating a damping force to hand movementcalculation, as in block 512.

An image input can include receiving images, as in block 522. In oneexample, the images can include images of hand and cutting tool imagesfrom a wrist worn smart device camera. The device can capture theimages, as in block 520. In another example, the received images caninclude color images which are extracted, for example, from a wrist wornsmart device camera to further enhance image recognition.

The computer can be used for estimating color based hand movement, as inblock 524. The estimation can include using a distance moved by a toolhand, that is, the hand holding the tool, and a tool hand movementvelocity. The method 500 includes the image input being received andused in the estimating of the damping force, as in block 512.

A data input can include receiving accelerometer readings of themovement, for example, a cutting process, as in block 532. Such data canbe generated from accelerometer readings, as in block 530. The distancemoved by a free hand and a free hand movement velocity can be determinedusing the accelerometer reading and the method can use the accelerometerestimation in the estimation of damping force to hand movement as inblock 512.

The computer can be used for determining accelerometer-based handmovement estimations, as in block 534.

The computer can be used for computing an audio notification, which canbe based on damping force and hand velocities. The audio notification oroutput can include a notification playout having and amplitude andfrequency, as in block 516.

The computer can be used for initiating or generating the audio output540 using a speaker in a wearable device or mobile device.

Further referring to FIG. 5 in general, the method 500 includes, forexample, a smart device is worn on a free hand wrist. From input sensordata color images, which can include skin-color images from camera (canbe obtained from wrist worn smart device camera (e.g., smart watch)),audio and accelerometer samples following variables can be estimatedincluding relative distance and velocity of the hands, and cut depth andvelocity.

The damping force can be calculated, given the distance, speed,acceleration estimates, sensitivity, and safety threshold of the systemfor hand(s) movement using the spring-mass-damper model (e.g., aspring-mass-damper model). Then, audio notification amplitude andfrequency of playout commensurate with a damping force and velocity ofthe hand(s), is computed. Finally, audio notifications with the aboveamplitude and frequency are played out or broadcast.

In one example, an audio-based hand movement estimation can include aDistance (Dt) and Velocity (Vt) of a tool hand which can be estimatedusing techniques using a Doppler effect, or a Time-of-arrival technique.Cut depth (Dd) and velocity (Vd) of the tool can be estimated using thespread-width and rate of change in spread, of the audio sample. Ingeneral, the wider the spread, the deeper the cut/chop.

In another example, a color based hand movement estimation can include aone-time calibration performed as follows: Create a database of thefollowing by sampling images at different distances between free andtool hands: for example: <Sratio=(Number of skin color pixels/Number oftotal pixels), D=distance>.

Then, apply linear regression on the above database to learn thefunction f(Sratio)=D Distance (Dt) and Velocity (Vt) of tool hand fromfree hand can be estimated as follows:

Compute Sratio for input image;

Estimate current distance from regression function:Current_Dt=f(Sratio);

Let, T be time elapsed since last sample and dprev previous estimate,Current_Vt=[(Current_Dt−Previous_Dt)/T];

Positive Current_V_(t) means tool hand is farther away, negativeCurrent_V_(t) means tool head closer than before.

In another example, an accelerometer based hand movement estimation caninclude Distance (Df) and Velocity (Vf) of free hand from tool hand canbe estimated taking double summation and single summation over time ofthe accelerometer values from the smart device sensor.

Referring to FIG. 6A, as the right hand 606 moves closer to left hand604 while handling an object 608, the audio notification (for example,an audio “ping” or “ting”) can become fast paced and louder. The graph612 depicts motion along an axis.

Referring to FIG. 6B, a graph 620 shows time along an x-axis andamplitude along a y-axis. The graph shows a notification waveform.

Referring to FIG. 7A, shows the right hand 606 moving closer to lefthand 604 while handling the object 608, in one example, a graph 712shows a visual indicator, for instance a yellow bar 714, which candenote too far for the left hand with respect to the right hand. As theleft hand moves away from right hand, an audio notification can beplayed.

Referring to FIG. 7B, a graph 720 shows a notification waveform along atime x-axis and an amplitude along a y-axis.

Referring to FIG. 8A, as the right hand 606 moves down-up, the audionotification (“tongs”) are played out, to warn user that the tool ismoving out of a range and that they should keep their left hand safelyaway.

Referring to FIG. 8B, a graph 820 shows a notification waveform along atime x-axis and an amplitude along a y-axis.

Referring to FIG. 9 . in another example, shows the right hand 606moving closer to left hand 604 while handling the object 608, in oneexample, a graph 912 shows a visual indicator, for instance a yellow bar914, which can denote too far for the left hand with respect to theright hand. An estimation of damping force for hand movement can includedamping force being computed using a spring-mass-damper model. Animaginary spring 904 is shown in relation to an imaginary damper 906.The spring-mass-damper model can include (Df, Dt, Vf, Vt, A) as inputsaveraged from previous steps, sensitivity input (K), safety threshold(L0) and tool hand mass (M), as follows:

Fd=−[M×A+K×D]

M: Mass of the tool hand (LH in the figure)

A: Acceleration of the tool hand

L0: Ideal safe distance between the hands

D=(L0−Df+Dt): Distance between the hands

K: Spring constant

In another example, computing an audio notification playout amplitudeand frequency including amplitude and playout frequency of the audionotification proportionally increasing with damping force (F_(d)) andhand velocity horizontal/depth (V):

A=A _(min)+[(A _(max) −A _(min))/(F _(max) −F _(min))]×F _(d)

A=A _(min)+[(A _(max) −A _(min))/(Max_D _(d)−Min_D _(d))]×D _(d)

f=f _(min)+[(f _(max) −f _(min))/(V _(max) −V _(min))]×V

F_(min): Minimum damping force

F_(max): Minimum damping force

Max_D_(d): Maximum cut depth

Min_D_(d): Minimum cut depth

V_(min): Minimum hand horizontal/depth velocity

V_(max): Maximum hand horizontal/depth velocity

A_(min): Minimum audio amplitude

A_(max): Minimum audio amplitude

f_(min): Minimum audio frequency

f_(min): Maximum audio frequency

Finally, the audio notification with amplitude A and frequency f isplayed out through the wearable device speaker.

The same technique could also be used to offer smart suggestions forbetter slicing. Since, a system could get the length of the object(under cut) and cut hit duration, it could predict the slice parametersalso (i.e., thickness of the slice). These parameters could be matchedto standard slice parameters and feedback could be given to end-user.Feedback could also be given through audio. For example, type of soundcould suggest the slice efficiency.

Thereby, the present disclosure can include a smart-watch based systemfor enabling adaptive audio based hand safety notifications duringcutting/chopping using sharp instruments. The system can include thefollowing operations. Enabling tracking using a combination of camera,audio and accelerometer sensor available on the wrist worn smart device.Applying spring-mass-damper system to model close-in, move-outinteractions of the hand using the above tracked data. Enable model(previous point) driven adaptation of amplitude and frequency of theaudio notification. Higher frequency and amplitude if hands (free handand tool hand) are closing-in. Lower frequency and amplitude when handsare moving-away. Deriving slice parameters and adapt the audio based onrequired slice cut efficiencies. This helps user cut slices withrequired thickness.

Thus, one embodiment of the present disclosure enables using asmart-watch. A system models close-in, move-out interactions by applyingspring-mass-damper system on vision and motion-sensor signals insmart-watch. The models are then executed in the smart-watch leveragingits dynamic sensor data to provide runtime audio feedback to users.

Dampening force can vary based on a hands proximity for different items.The length of different items can be factored into the equation by asetting. This parameter will influence the dampening force. Thethreshold can be changed based on the item that is being cut usingthresholds which can be pre-configured by crowd-sourced methods as inthe case of mechanical tools. Sample of heuristic collection andcalculation process can include leveraging videos to label safe cuttingpatterns. Analyze the cutting actions in a semi-supervised manner. Noisepollution can be controlled by using ambient noise signatures could belearnt over time and eliminated. Many speech-assistants, conversationalagents use several techniques like microphone arrays (miniaturized),background noise learning techniques to achieve this.

In another example, related to using hand motions for slicing, a similartechnique could also be used to offer smart suggestions for betterslicing. Since the system could get the length of the object (under cut)and cut hit duration, it could predict the slice parameters also (i.e.thickness of the slice). These parameters could be matched to standardslice parameters and feedback could be given to end-user. The suggestioncan take into account these variables to account for diversity ofobjects that are to be chopped. The objects to be cut can be ofdifferent sizes. Some are round while others are long while others mightbe larger. For objects which are round & big, it normally involves onelarge cut and then smaller cuts. Since this suggestion is based only onlength of object & not the circumference & volume. The force for slicecan be function of these parameters. Feedback could also be giventhrough audio. For example, a type of sound could suggest the sliceefficiency.

The following are examples in case of special circumstances for users,such as, visually impairment. Embodiments of the present disclosure canfacilitate a safe cutting process through pro-active auditory cues. If aperson has auditory impairment, embodiment for the present disclosurecan initiate and implement haptic cues which facilitate notificationsfor users with hearing difficulties.

In one example, vibrational haptic devices can be generated by awearable device to evoke a movement. The vibrational stimulus will alsodetermine if a hand has become numb by using computer vision to detectan expected movement.

A haptic device, for example, can be worn in clothes or a glove. Awearable device including the method and system of the presentdisclosure can enable the evoking of rapid reflexive correctionmovements that optimize safety. The method and system can use the deviceto learn what vibrational frequency and amplitude to use for a specificuser based on expected movements. The method and system can use thedevice to learn to avoid numbness of nerves. The method and system canuse a device to enable proprioception movement masking by usingvibrational stimulus on muscle spindles to indicate a certain movementwas already being performed.

More Examples and Embodiments

Operational blocks and system components shown in one or more of thefigures may be similar to operational blocks and system components inother figures. The diversity of operational blocks and system componentsdepict example embodiments and aspects according to the presentdisclosure. For example, methods shown are intended as exampleembodiments which can include aspects/operations shown and discussedpreviously in the present disclosure, and in one example, continuingfrom a previous method shown in another flow chart.

Additional Examples and Embodiments

In the embodiment of the present disclosure shown in FIGS. 1 and 2 , acomputer can be part of a remote computer or a remote server, forexample, remote server 1100 (FIG. 10 ). In another example, the computer131 can be part of a control system 170 and provide execution of thefunctions of the present disclosure. In another embodiment, a computercan be part of a mobile device and provide execution of the functions ofthe present disclosure. In still another embodiment, parts of theexecution of functions of the present disclosure can be shared betweenthe control system computer and the mobile device computer, for example,the control system function as a back end of a program or programsembodying the present disclosure and the mobile device computerfunctioning as a front end of the program or programs.

The computer can be part of the mobile device, or a remote computercommunicating with the mobile device. In another example, a mobiledevice and a remote computer can work in combination to implement themethod of the present disclosure using stored program code orinstructions to execute the features of the method(s) described herein.In one example, the device 130 can include a computer 131 having aprocessor 132 and a storage medium 134 which stores an application 135,and the computer includes a display 138. The application can incorporateprogram instructions for executing the features of the presentdisclosure using the processor 132. In another example, the mobiledevice application or computer software can have program instructionsexecutable for a front end of a software application incorporating thefeatures of the method of the present disclosure in programinstructions, while a back end program or programs 174, of the softwareapplication, stored on the computer 172 of the control system 170communicates with the mobile device computer and executes other featuresof the method. The control system 170 and the device (e.g., mobiledevice or computer) 130 can communicate using a communications network160, for example, the Internet.

Thereby, the method 100 according to an embodiment of the presentdisclosure, can be incorporated in one or more computer programs or anapplication 135 stored on an electronic storage medium 134, andexecutable by the processor 132, as part of the computer on mobiledevice. For example, a mobile device can communicate with the controlsystem 170, and in another example, a device such as a video feed devicecan communicate directly with the control system 170. Other users (notshown) may have similar mobile devices which communicate with thecontrol system similarly. The application can be stored, all or in part,on a computer or a computer in a mobile device and at a control systemcommunicating with the mobile device, for example, using thecommunications network 160, such as the Internet. It is envisioned thatthe application can access all or part of program instructions toimplement the method of the present disclosure. The program orapplication can communicate with a remote computer system via acommunications network 160 (e.g., the Internet) and access data, andcooperate with program(s) stored on the remote computer system. Suchinteractions and mechanisms are described in further detail herein andreferred to regarding components of a computer system, such as computerreadable storage media, which are shown in one embodiment in FIG. 10 anddescribed in more detail in regards thereto referring to one or morecomputer systems 1010.

Thus, in one example, a control system 170 is in communication with thecomputer 131 or device 130, and the computer can include the applicationor software 135. The computer 131, or a computer in a mobile device 130communicates with the control system 170 using the communicationsnetwork 160.

In another example, the control system 170 can have a front-end computerbelonging to one or more users, and a back-end computer embodied as thecontrol system.

Also, referring to FIG. 1 , a device 130 can include a computer 131,computer readable storage medium 134, and operating systems, and/orprograms, and/or a software application 135, which can include programinstructions executable using a processor 132. These features are shownherein in FIG. 1 , and other similar components and features are also inan embodiment of a computer system shown in FIG. 10 referring to acomputer system 1010, which may include one or more computer components.

The method according to the present disclosure, can include a computerfor implementing the features of the method, according to the presentdisclosure, as part of a control system. In another example, a computeras part of a control system can work in corporation with a mobile devicecomputer in concert with communication system for implementing thefeatures of the method according to the present disclosure. In anotherexample, a computer for implementing the features of the method can bepart of a mobile device and thus implement the method locally.

Specifically, regarding the control system 170, a device(s) 130, or inone example devices which can belong to one or more users, can be incommunication with the control system 170 via the communications network160. In the embodiment of the control system shown in FIG. 1 , thecontrol system 170 includes a computer 172 communicating with a database176 and one or more programs 174 stored on a computer readable storagemedium 173. In the embodiment of the disclosure shown in FIG. 1 , thedevice 130 communicates with the control system 170 and the one or moreprograms 174 stored on a computer readable storage medium 173. Thecontrol system includes the computer 172 having a processor 175, whichalso has access to the database 176.

The control system 170 can include a storage medium 180 for maintaininga registration 182 of users and their devices for analysis of the audioinput. Such registration can include user profiles 183, which caninclude user data supplied by the users in reference to registering andsetting-up an account. In an embodiment, the method and system whichincorporates the present disclosure includes the control system(generally referred to as the back-end) in combination and cooperationwith a front end of the method and system, which can be the application135. In one example, the application 135 is stored on a device, forexample, a computer or device on location, and can access data andadditional programs at a back end of the application, e.g., controlsystem 170.

The control system can also be part of a software applicationimplementation, and/or represent a software application having afront-end user part and a back-end part providing functionality. In anembodiment, the method and system which incorporates the presentdisclosure includes the control system (which can be generally referredto as the back-end of the software application which incorporates a partof the method and system of an embodiment of the present application) incombination and cooperation with a front end of the software applicationincorporating another part of the method and system of the presentapplication at the device, as in the example shown in FIG. 1 of a device130 and computer 131 having the application 135. The application 135 isstored on the device or computer and can access data and additionalprograms at the back end of the application, for example, in theprogram(s) 174 stored in the control system 170.

The program(s) 174 can include, all or in part, a series of executablesteps for implementing the method of the present disclosure. A program,incorporating the present method, can be all or in part stored in thecomputer readable storage medium on the control system or, in all or inpart, on a computer or device 130. It is envisioned that the controlsystem 170 can not only store the profile of users, but in oneembodiment, can interact with a website for viewing on a display of adevice such as a mobile device, or in another example the Internet, andreceive user input related to the method and system of the presentdisclosure. It is understood that FIG. 1 depicts one or more profiles183, however, the method can include multiple profiles, users,registrations, etc. It is envisioned that a plurality of users or agroup of users can register and provide profiles using the controlsystem for use according to the method and system of the presentdisclosure.

Still Further Embodiments and Examples

It is understood that the features shown in some of the FIGS., forexample block diagrams, are functional representations of features ofthe present disclosure. Such features are shown in embodiments of thesystems and methods of the present disclosure for illustrative purposesto clarify the functionality of features of the present disclosure.

The methods and systems of the present disclosure can include a seriesof operation blocks for implementing one or more embodiments accordingto the present disclosure. In some examples, operational blocks of oneor more FIGS. may be similar to operational blocks shown in anotherfigure. A method shown in one FIG. may be another example embodimentwhich can include aspects/operations shown in another FIG. and discussedpreviously.

Additional Embodiments and Examples

Account data, for instance, including profile data related to a user,and any data, personal or otherwise, can be collected and stored, forexample, in the control system 170. It is understood that such datacollection is done with the knowledge and consent of a user, and storedto preserve privacy, which is discussed in more detail below. Such datacan include personal data, and data regarding personal items.

In one example a user can register 182 have an account 181 with a userprofile 183 on a control system 170, which is discussed in more detailbelow. For example, data can be collected using techniques as discussedabove, for example, using cameras, and data can be uploaded to a userprofile by the user. A user can include, for example, a corporateentity, or department of a business, or a homeowner, or any end user, ahuman operator, or a robotic device, or other personnel of a business.

Regarding collection of data with respect to the present disclosure,such uploading or generation of profiles is voluntary by the one or moreusers, and thus initiated by and with the approval of a user. Thereby, auser can opt-in to establishing an account having a profile according tothe present disclosure. Similarly, data received by the system orinputted or received as an input is voluntary by one or more users, andthus initiated by and with the approval of the user. Thereby, a user canopt-in to input data according to the present disclosure. Such userapproval also includes a user's option to cancel such profile oraccount, and/or input of data, and thus opt-out, at the user'sdiscretion, of capturing communications and data. Further, any datastored or collected is understood to be intended to be securely storedand unavailable without authorization by the user, and not available tothe public and/or unauthorized users. Such stored data is understood tobe deleted at the request of the user and deleted in a secure manner.Also, any use of such stored data is understood to be, according to thepresent disclosure, only with the user's authorization and consent.

In one or more embodiments of the present invention, a user(s) canopt-in or register with a control system, voluntarily providing dataand/or information in the process, with the user's consent andauthorization, where the data is stored and used in the one or moremethods of the present disclosure. Also, a user(s) can register one ormore user electronic devices for use with the one or more methods andsystems according to the present disclosure. As part of a registration,a user can also identify and authorize access to one or more activitiesor other systems (e.g., audio and/or video systems). Such opt-in ofregistration and authorizing collection and/or storage of data isvoluntary and a user may request deletion of data (including a profileand/or profile data), un-registering, and/or opt-out of anyregistration. It is understood that such opting-out includes disposal ofall data in a secure manner. A user interface can also allow a user oran individual to remove all their historical data.

Other Additional Embodiments and Examples

In one example, Artificial Intelligence (AI) can be used, all or inpart, for generating a model or a learning model as discussed herein inembodiments of the present disclosure. An Artificial Intelligence (AI)System can include machines, computer, and computer programs which aredesigned to be intelligent or mirror intelligence. Such systems caninclude computers executing algorithms. AI can include machine learningand deep learning. For example, deep learning can include neuralnetworks. An AI system can be cloud based, that is, using a cloud-basedcomputing environment having computing resources.

In another example, the control system 170 can be all or part of anArtificial Intelligence (AI) system. For example, the control system canbe one or more components of an AI system.

It is also understood that the method 100 according to an embodiment ofthe present disclosure, can be incorporated into (ArtificialIntelligence) AI devices, components or be part of an AI system, whichcan communicate with respective AI systems and components, andrespective AI system platforms. Thereby, such programs or an applicationincorporating the method of the present disclosure, as discussed above,can be part of an AI system. In one embodiment according to the presentinvention, it is envisioned that the control system can communicate withan AI system, or in another example can be part of an AI system. Thecontrol system can also represent a software application having afront-end user part and a back-end part providing functionality, whichcan in one or more examples, interact with, encompass, or be part oflarger systems, such as an AI system. In one example, an AI device canbe associated with an AI system, which can be all or in part, a controlsystem and/or a content delivery system, and be remote from an AIdevice. Such an AI system can be represented by one or more serversstoring programs on computer readable medium which can communicate withone or more AI devices. The AI system can communicate with the controlsystem, and in one or more embodiments, the control system can be all orpart of the AI system or vice versa.

It is understood that as discussed herein, a download or downloadabledata can be initiated using a voice command or using a mouse, touchscreen, etc. In such examples a mobile device can be user initiated, oran AI device can be used with consent and permission of users. Otherexamples of AI devices include devices which include a microphone,speaker, and can access a cellular network or mobile network, acommunications network, or the Internet, for example, a vehicle having acomputer and having cellular or satellite communications, or in anotherexample, IoT (Internet of Things) devices, such as appliances, havingcellular network or Internet access.

Further Discussion Regarding Examples and Embodiments

It is understood that a set or group is a collection of distinct objectsor elements. The objects or elements that make up a set or group can beanything, for example, numbers, letters of the alphabet, other sets, anumber of people or users, and so on. It is further understood that aset or group can be one element, for example, one thing or a number, inother words, a set of one element, for example, one or more users orpeople or participants. It is also understood that machine and deviceare used interchangeable herein to refer to machine or devices in one ormore AI ecosystems or environments.

The descriptions of the various embodiments of the present inventionhave been presented for purposes of illustration, but are not intendedto be exhaustive or limited to the embodiments disclosed. Likewise,examples of features or functionality of the embodiments of thedisclosure described herein, whether used in the description of aparticular embodiment, or listed as examples, are not intended to limitthe embodiments of the disclosure described herein, or limit thedisclosure to the examples described herein. Such examples are intendedto be examples or exemplary, and non-exhaustive. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the describedembodiments. The terminology used herein was chosen to best explain theprinciples of the embodiments, the practical application or technicalimprovement over technologies found in the marketplace, or to enableothers of ordinary skill in the art to understand the embodimentsdisclosed herein.

Further Additional Examples and Embodiments

Referring to FIG. 10 , an embodiment of system or computer environment1000, according to the present disclosure, includes a computer system1010 shown in the form of a generic computing device. The method 100,for example, may be embodied in a program 1060, including programinstructions, embodied on a computer readable storage device, or acomputer readable storage medium, for example, generally referred to ascomputer memory 1030 and more specifically, computer readable storagemedium 1050. Such memory and/or computer readable storage media includesnon-volatile memory or non-volatile storage, also known and referred tonon-transient computer readable storage media, or non-transitorycomputer readable storage media. For example, such non-volatile memorycan also be disk storage devices, including one or more hard drives. Forexample, memory 1030 can include storage media 1034 such as RAM (RandomAccess Memory) or ROM (Read Only Memory), and cache memory 1038. Theprogram 1060 is executable by the processor 1020 of the computer system1010 (to execute program steps, code, or program code). Additional datastorage may also be embodied as a database 1110 which includes data1114. The computer system 1010 and the program 1060 are genericrepresentations of a computer and program that may be local to a user,or provided as a remote service (for example, as a cloud based service),and may be provided in further examples, using a website accessibleusing the communications network 1200 (e.g., interacting with a network,the Internet, or cloud services). It is understood that the computersystem 1010 also generically represents herein a computer device or acomputer included in a device, such as a laptop or desktop computer,etc., or one or more servers, alone or as part of a datacenter. Thecomputer system can include a network adapter/interface 1026, and aninput/output (I/O) interface(s) 1022. The I/O interface 1022 allows forinput and output of data with an external device 1074 that may beconnected to the computer system. The network adapter/interface 1026 mayprovide communications between the computer system a network genericallyshown as the communications network 1200.

The computer 1010 may be described in the general context of computersystem-executable instructions, such as program modules, being executedby a computer system. Generally, program modules may include routines,programs, objects, components, logic, data structures, and so on thatperform particular tasks or implement particular abstract data types.The method steps and system components and techniques may be embodied inmodules of the program 1060 for performing the tasks of each of thesteps of the method and system. The modules are generically representedin the figure as program modules 1064. The program 1060 and programmodules 1064 can execute specific steps, routines, sub-routines,instructions or code, of the program.

The method of the present disclosure can be run locally on a device suchas a mobile device, or can be run a service, for instance, on the server1100 which may be remote and can be accessed using the communicationsnetwork 1200. The program or executable instructions may also be offeredas a service by a provider. The computer 1010 may be practiced in adistributed cloud computing environment where tasks are performed byremote processing devices that are linked through a communicationsnetwork 1200. In a distributed cloud computing environment, programmodules may be located in both local and remote computer system storagemedia including memory storage devices.

More specifically, the system or computer environment 1000 includes thecomputer system 1010 shown in the form of a general-purpose computingdevice with illustrative periphery devices. The components of thecomputer system 1010 may include, but are not limited to, one or moreprocessors or processing units 1020, a system memory 1030, and a bus1014 that couples various system components including system memory 1030to processor 1020.

The bus 1014 represents one or more of any of several types of busstructures, including a memory bus or memory controller, a peripheralbus, an accelerated graphics port, and a processor or local bus usingany of a variety of bus architectures. By way of example, and notlimitation, such architectures include Industry Standard Architecture(ISA) bus, Micro Channel Architecture (MCA) bus, Enhanced ISA (EISA)bus, Video Electronics Standards Association (VESA) local bus, andPeripheral Component Interconnects (PCI) bus.

The computer 1010 can include a variety of computer readable media. Suchmedia may be any available media that is accessible by the computer 1010(e.g., computer system, or server), and can include both volatile andnon-volatile media, as well as removable and non-removable media.Computer memory 1030 can include additional computer readable media inthe form of volatile memory, such as random access memory (RAM) 1034,and/or cache memory 1038. The computer 1010 may further include otherremovable/non-removable, volatile/non-volatile computer storage media,in one example, portable computer readable storage media 1072. In oneembodiment, the computer readable storage medium 1050 can be providedfor reading from and writing to a non-removable, non-volatile magneticmedia. The computer readable storage medium 1050 can be embodied, forexample, as a hard drive. Additional memory and data storage can beprovided, for example, as the storage system 1110 (e.g., a database) forstoring data 1114 and communicating with the processing unit 1020. Thedatabase can be stored on or be part of a server 1100. Although notshown, a magnetic disk drive for reading from and writing to aremovable, non-volatile magnetic disk (e.g., a “floppy disk”), and anoptical disk drive for reading from or writing to a removable,non-volatile optical disk such as a CD-ROM, DVD-ROM or other opticalmedia can be provided. In such instances, each can be connected to bus1014 by one or more data media interfaces. As will be further depictedand described below, memory 1030 may include at least one programproduct which can include one or more program modules that areconfigured to carry out the functions of embodiments of the presentinvention.

The method(s) described in the present disclosure, for example, may beembodied in one or more computer programs, generically referred to as aprogram 1060 and can be stored in memory 1030 in the computer readablestorage medium 1050. The program 1060 can include program modules 1064.The program modules 1064 can generally carry out functions and/ormethodologies of embodiments of the invention as described herein. Theone or more programs 1060 are stored in memory 1030 and are executableby the processing unit 1020. By way of example, the memory 1030 maystore an operating system 1052, one or more application programs 1054,other program modules, and program data on the computer readable storagemedium 1050. It is understood that the program 1060, and the operatingsystem 1052 and the application program(s) 1054 stored on the computerreadable storage medium 1050 are similarly executable by the processingunit 1020. It is also understood that the application 1054 andprogram(s) 1060 are shown generically, and can include all of, or bepart of, one or more applications and program discussed in the presentdisclosure, or vice versa, that is, the application 1054 and program1060 can be all or part of one or more applications or programs whichare discussed in the present disclosure. It is also understood that acontrol system 170, communicating with a computer system, can includeall or part of the computer system 1010 and its components, and/or thecontrol system can communicate with all or part of the computer system1010 and its components as a remote computer system, to achieve thecontrol system functions described in the present disclosure. Thecontrol system function, for example, can include storing, processing,and executing software instructions to perform the functions of thepresent disclosure. It is also understood that the one or more computersor computer systems shown in FIG. 1 similarly can include all or part ofthe computer system 1010 and its components, and/or the one or morecomputers can communicate with all or part of the computer system 1010and its components as a remote computer system, to achieve the computerfunctions described in the present disclosure.

In an embodiment according to the present disclosure, one or moreprograms can be stored in one or more computer readable storage mediasuch that a program is embodied and/or encoded in a computer readablestorage medium. In one example, the stored program can include programinstructions for execution by a processor, or a computer system having aprocessor, to perform a method or cause the computer system to performone or more functions. For example, in one embedment according to thepresent disclosure, a program embodying a method is embodied in, orencoded in, a computer readable storage medium, which includes and isdefined as, a non-transient or non-transitory computer readable storagemedium. Thus, embodiments or examples according to the presentdisclosure, of a computer readable storage medium do not include asignal, and embodiments can include one or more non-transient ornon-transitory computer readable storage mediums. Thereby, in oneexample, a program can be recorded on a computer readable storage mediumand become structurally and functionally interrelated to the medium.

The computer 1010 may also communicate with one or more external devices1074 such as a keyboard, a pointing device, a display 1080, etc.; one ormore devices that enable a user to interact with the computer 1010;and/or any devices (e.g., network card, modem, etc.) that enables thecomputer 1010 to communicate with one or more other computing devices.Such communication can occur via the Input/Output (I/O) interfaces 1022.A power supply 1090 can also connect to the computer using an electricalpower supply interface (not shown). Still yet, the computer 1010 cancommunicate with one or more networks 1200 such as a local area network(LAN), a general wide area network (WAN), and/or a public network (e.g.,the Internet) via network adapter/interface 1026. As depicted, networkadapter 1026 communicates with the other components of the computer 1010via bus 1014. It should be understood that although not shown, otherhardware and/or software components could be used in conjunction withthe computer 1010. Examples, include, but are not limited to: microcode,device drivers 1024, redundant processing units, external disk drivearrays, RAID systems, tape drives, and data archival storage systems,etc.

It is understood that a computer or a program running on the computer1010 may communicate with a server, embodied as the server 1100, via oneor more communications networks, embodied as the communications network1200. The communications network 1200 may include transmission media andnetwork links which include, for example, wireless, wired, or opticalfiber, and routers, firewalls, switches, and gateway computers. Thecommunications network may include connections, such as wire, wirelesscommunication links, or fiber optic cables. A communications network mayrepresent a worldwide collection of networks and gateways, such as theInternet, that use various protocols to communicate with one another,such as Lightweight Directory Access Protocol (LDAP), Transport ControlProtocol/Internet Protocol (TCP/IP), Hypertext Transport Protocol(HTTP), Wireless Application Protocol (WAP), etc. A network may alsoinclude a number of different types of networks, such as, for example,an intranet, a local area network (LAN), or a wide area network (WAN).

In one example, a computer can use a network which may access a websiteon the Web (World Wide Web) using the Internet. In one embodiment, acomputer 1010, including a mobile device, can use a communicationssystem or network 1200 which can include the Internet, or a publicswitched telephone network (PSTN) for example, a cellular network. ThePSTN may include telephone lines, fiber optic cables, microwavetransmission links, cellular networks, and communications satellites.The Internet may facilitate numerous searching and texting techniques,for example, using a cell phone or laptop computer to send queries tosearch engines via text messages (SMS), Multimedia Messaging Service(MMS) (related to SMS), email, or a web browser. The search engine canretrieve search results, that is, links to websites, documents, or otherdownloadable data that correspond to the query, and similarly, providethe search results to the user via the device as, for example, a webpage of search results.

Still Further Additional Examples and Embodiments

Referring to FIG. 11 , an example system 1500 for use with theembodiments of the present disclosure is depicted. The system 1500includes a plurality of components and elements connected via a systembus 1504. At least one processor (CPU) 1510, is connected to othercomponents via the system bus 1504. A cache 1570, a Read Only Memory(ROM) 1512, a Random Access Memory (RAM) 1514, an input/output (I/O)adapter 1520, a sound adapter 1530, a network adapter 1540, a userinterface adapter 1552, a display adapter 1560 and a display device1562, are also operatively coupled to the system bus 1504 of the system1500. An AR device 1580 can also be operatively coupled to the bus 1504.An AI enabled robotic device and control system 1580 can also beoperatively coupled to the bus 1504. Such a robot and control system1580 can incorporate all or part of embodiments of the presentdisclosure and discussed hereinbefore. An artificial intelligence (AI)system 1575 or an AI ecosystem can also be operatively coupled to thebus 1504. A power supply 1595 can also be operatively connected to thebus 1504 for providing power to components and for functions accordingto the present disclosure. An augmented reality (AR) device 1590 canalso be operatively connected to the bus 1504 for providing augmentedreality output to a wearable augmented reality device, such as ARglasses or an AR headset.

One or more storage devices 1522 are operatively coupled to the systembus 1504 by the I/O adapter 1520. The storage device 1522, for example,can be any of a disk storage device (e.g., a magnetic or optical diskstorage device), a solid state magnetic device, and so forth. Thestorage device 1522 can be the same type of storage device or differenttypes of storage devices. The storage device can include, for example,but not limited to, a hard drive or flash memory and be used to storeone or more programs 1524 or applications 1526. The programs andapplications are shown as generic components and are executable usingthe processor 1510. The program 1524 and/or application 1526 can includeall of, or part of, programs or applications discussed in the presentdisclosure, as well vice versa, that is, the program 1524 and theapplication 1526 can be part of other applications or program discussedin the present disclosure.

The system 1500 can include the control system 170 which is part of thesystem 100 (described in further detail hereinbefore) and cancommunicate with the system bus independently or as part of the system100, and thus can communicate with the other components of the system1500 via the system bus. In one example, the storage device 1522, viathe system bus, can communicate with the control system 170 which hasvarious functions as described in the present disclosure.

In one aspect, a speaker 1532 is operatively coupled to system bus 1504by the sound adapter 1530. A transceiver 1542 is operatively coupled tosystem bus 1504 by the network adapter 1540. A display 1562 isoperatively coupled to the system bus 1504 by the display adapter 1560.

In another aspect, one or more user input devices 1550 are operativelycoupled to the system bus 1504 by the user interface adapter 1552. Theuser input devices 1550 can be, for example, any of a keyboard, a mouse,a keypad, an image capture device, a motion sensing device, amicrophone, a device incorporating the functionality of at least two ofthe preceding devices, and so forth. Other types of input devices canalso be used, while maintaining the spirit of the present invention. Theuser input devices 1550 can be the same type of user input device ordifferent types of user input devices. The user input devices 1550 areused to input and output information to and from the system 1500.

Other Aspects and Examples

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures of the presentdisclosure illustrate the architecture, functionality, and operation ofpossible implementations of systems, methods, and computer programproducts according to various embodiments of the present invention. Inthis regard, each block in the flowchart or block diagrams may representa module, segment, or portion of instructions, which comprises one ormore executable instructions for implementing the specified logicalfunction(s). In some alternative implementations, the functions noted inthe blocks may occur out of the order noted in the Figures. For example,two blocks shown in succession may, in fact, be accomplished as onestep, executed concurrently, substantially concurrently, in a partiallyor wholly temporally overlapping manner, or the blocks may sometimes beexecuted in the reverse order, depending upon the functionalityinvolved. It will also be noted that each block of the block diagramsand/or flowchart illustration, and combinations of blocks in the blockdiagrams and/or flowchart illustration, can be implemented by specialpurpose hardware-based systems that perform the specified functions oracts or carry out combinations of special purpose hardware and computerinstructions.

Additional Aspects and Examples

It is to be understood that although this disclosure includes a detaileddescription on cloud computing, implementation of the teachings recitedherein are not limited to a cloud computing environment. Rather,embodiments of the present invention are capable of being implemented inconjunction with any other type of computing environment now known orlater developed.

Cloud computing is a model of service delivery for enabling convenient,on-demand network access to a shared pool of configurable computingresources (e.g., networks, network bandwidth, servers, processing,memory, storage, applications, virtual machines, and services) that canbe rapidly provisioned and released with minimal management effort orinteraction with a provider of the service. This cloud model may includeat least five characteristics, at least three service models, and atleast four deployment models.

Characteristics are as follows:

On-demand self-service: a cloud consumer can unilaterally provisioncomputing capabilities, such as server time and network storage, asneeded automatically without requiring human interaction with theservice's provider.

Broad network access: capabilities are available over a network andaccessed through standard mechanisms that promote use by heterogeneousthin or thick client platforms (e.g., mobile phones, laptops, and PDAs).

Resource pooling: the provider's computing resources are pooled to servemultiple consumers using a multi-tenant model, with different physicaland virtual resources dynamically assigned and reassigned according todemand. There is a sense of location independence in that the consumergenerally has no control or knowledge over the exact location of theprovided resources but may be able to specify location at a higher levelof abstraction (e.g., country, state, or datacenter).

Rapid elasticity: capabilities can be rapidly and elasticallyprovisioned, in some cases automatically, to quickly scale out andrapidly released to quickly scale in. To the consumer, the capabilitiesavailable for provisioning often appear to be unlimited and can bepurchased in any quantity at any time.

Measured service: cloud systems automatically control and optimizeresource use by leveraging a metering capability at some level ofabstraction appropriate to the type of service (e.g., storage,processing, bandwidth, and active user accounts). Resource usage can bemonitored, controlled, and reported, providing transparency for both theprovider and consumer of the utilized service.

Service Models are as follows:

Software as a Service (SaaS): the capability provided to the consumer isto use the provider's applications running on a cloud infrastructure.The applications are accessible from various client devices through athin client interface such as a web browser (e.g., web-based e-mail).The consumer does not manage or control the underlying cloudinfrastructure including network, servers, operating systems, storage,or even individual application capabilities, with the possible exceptionof limited user-specific application configuration settings.

Platform as a Service (PaaS): the capability provided to the consumer isto deploy onto the cloud infrastructure consumer-created or acquiredapplications created using programming languages and tools supported bythe provider. The consumer does not manage or control the underlyingcloud infrastructure including networks, servers, operating systems, orstorage, but has control over the deployed applications and possiblyapplication hosting environment configurations.

Infrastructure as a Service (IaaS): the capability provided to theconsumer is to provision processing, storage, networks, and otherfundamental computing resources where the consumer is able to deploy andrun arbitrary software, which can include operating systems andapplications. The consumer does not manage or control the underlyingcloud infrastructure but has control over operating systems, storage,deployed applications, and possibly limited control of select networkingcomponents (e.g., host firewalls).

Deployment Models are as follows:

Private cloud: the cloud infrastructure is operated solely for anorganization. It may be managed by the organization or a third party andmay exist on-premises or off-premises.

Community cloud: the cloud infrastructure is shared by severalorganizations and supports a specific community that has shared concerns(e.g., mission, security requirements, policy, and complianceconsiderations). It may be managed by the organizations or a third partyand may exist on-premises or off-premises.

Public cloud: the cloud infrastructure is made available to the generalpublic or a large industry group and is owned by an organization sellingcloud services.

Hybrid cloud: the cloud infrastructure is a composition of two or moreclouds (private, community, or public) that remain unique entities butare bound together by standardized or proprietary technology thatenables data and application portability (e.g., cloud bursting forload-balancing between clouds).

A cloud computing environment is service oriented with a focus onstatelessness, low coupling, modularity, and semantic interoperability.At the heart of cloud computing is an infrastructure that includes anetwork of interconnected nodes.

Referring now to FIG. 12 , illustrative cloud computing environment 2050is depicted. As shown, cloud computing environment 2050 includes one ormore cloud computing nodes 2010 with which local computing devices usedby cloud consumers, such as, for example, personal digital assistant(PDA) or cellular telephone 2054A, desktop computer 2054B, laptopcomputer 2054C, and/or automobile computer system 2054N may communicate.Nodes 2010 may communicate with one another. They may be grouped (notshown) physically or virtually, in one or more networks, such asPrivate, Community, Public, or Hybrid clouds as described hereinabove,or a combination thereof. This allows cloud computing environment 2050to offer infrastructure, platforms and/or software as services for whicha cloud consumer does not need to maintain resources on a localcomputing device. It is understood that the types of computing devices2054A-N shown in FIG. 12 are intended to be illustrative only and thatcomputing nodes 2010 and cloud computing environment 2050 cancommunicate with any type of computerized device over any type ofnetwork and/or network addressable connection (e.g., using a webbrowser).

Referring now to FIG. 13 , a set of functional abstraction layersprovided by cloud computing environment 2050 (FIG. 12 ) is shown. Itshould be understood in advance that the components, layers, andfunctions shown in FIG. 13 are intended to be illustrative only andembodiments of the invention are not limited thereto. As depicted, thefollowing layers and corresponding functions are provided:

Hardware and software layer 2060 includes hardware and softwarecomponents. Examples of hardware components include: mainframes 2061;RISC (Reduced Instruction Set Computer) architecture based servers 2062;servers 2063; blade servers 2064; storage devices 2065; and networks andnetworking components 2066. In some embodiments, software componentsinclude network application server software 2067 and database software2068.

Virtualization layer 2070 provides an abstraction layer from which thefollowing examples of virtual entities may be provided: virtual servers2071; virtual storage 2072; virtual networks 2073, including virtualprivate networks; virtual applications and operating systems 2074; andvirtual clients 2075.

In one example, management layer 2080 may provide the functionsdescribed below. Resource provisioning 2081 provides dynamic procurementof computing resources and other resources that are utilized to performtasks within the cloud computing environment. Metering and Pricing 2082provide cost tracking as resources are utilized within the cloudcomputing environment, and billing or invoicing for consumption of theseresources. In one example, these resources may include applicationsoftware licenses. Security provides identity verification for cloudconsumers and tasks, as well as protection for data and other resources.User portal 2083 provides access to the cloud computing environment forconsumers and system administrators. Service level management 2084provides cloud computing resource allocation and management such thatrequired service levels are met. Service Level Agreement (SLA) planningand fulfillment 2085 provide pre-arrangement for, and procurement of,cloud computing resources for which a future requirement is anticipatedin accordance with an SLA.

Workloads layer 2090 provides examples of functionality for which thecloud computing environment may be utilized. Examples of workloads andfunctions which may be provided from this layer include: mapping andnavigation 2091; software development and lifecycle management 2092;virtual classroom education delivery 2093; data analytics processing2094; transaction processing 2095; and Artificial Intelligence enabledfeedback for assisting a person in a physical task 2096, for example,providing feedback in response to spatial positioning between a user'shands and an object, using a wearable device and a computer, forinstance, a control system having a computer.

What is claimed is:
 1. A computer-implemented method for detectingspatial positioning between a user's hands and an object, using awearable device, to provide feedback to a user, comprising: receiving,at a computer, motion data of a person's body motion while performing anaction, the motion data including sensor data from sensors at alocation, the sensors detecting the person's body motion, and thesensors including a wearable device on the person's body; modeling,using the computer, the person's body motion using the motion data;determining, using the computer, a set of parameters for acceptablemotions based on an action risk assessment of the action; and initiatingfeedback, using the wearable device, to the person based on the person'sbody motion exceeding a body motion threshold based on the set ofparameters for the acceptable motions and the model.
 2. The method ofclaim 1, wherein the feedback is haptic feedback generated using thewearable device.
 3. The method of claim 1, further comprising:determining the body motion threshold by comparing the set of parametersfor the acceptable motion to the model of the person's body motions. 4.The method of claim 1, further comprising: determining the body motionthreshold by comparing the set of parameters for the acceptable motionto the model of the person's body motions, wherein the comparingincludes a risk assessment of the body motions based on the riskassessment of the action.
 5. The method of claim 1, further comprising:determining the body motion threshold by comparing the set of parametersfor the acceptable motion to the model of the person's body motions,wherein the comparing includes a risk assessment of the body motionsbased on the action risk assessment to determine a risk factor for aparticular motion.
 6. The method of claim 1, further comprising:determining the body motion threshold by comparing the set of parametersfor the acceptable motion to the model of the person's body motions,wherein the comparing includes a risk assessment of the body motionsbased on the action risk assessment to determine a risk factor for aparticular motion, wherein the body motion threshold is based on therisk factor for the particular motion.
 7. The method of claim 1, furthercomprising: determining the body motion threshold by comparing the setof parameters for the acceptable motion to the model of the person'sbody motions, wherein the comparing includes a risk assessment of thebody motions based on the action risk assessment to determine a riskfactor for a particular motion, wherein the body motion threshold isbased on the risk factor for the particular motion, wherein the bodymotion threshold does not exceed a risk factor variable.
 8. The methodof claim 7, wherein the risk factor variable is based on the set ofparameters for acceptable motions.
 9. The method of claim 1, wherein theinitiating of the feedback includes sending a communication to thewearable device.
 10. The method of claim 1, wherein the body motionincludes hand motions of the person.
 11. The method of claim 1, furthercomprising: sending a communication to the wearable device enablingwarnings from the wearable device when the user's hand motions exceed athreshold based on the set of the parameters for the acceptable motions.12. The method of claim 1, wherein the sensors at the location includinga combination of one or more site sensors and a wearable sensor.
 13. Themethod of claim 1, wherein the site sensors include a camera, an audiosensor or microphone device, and the wearable sensor includes anaccelerometer.
 14. The method of claim 1, wherein the wearable sensor isa wrist worn smart device or smartwatch.
 15. The method of claim 1,wherein the modeling of the person's body motion, including modeling auser's hand motions using the motion data.
 16. The method of claim 1,wherein the modeling of the person's body motion includes using themotion data and a spring-mass-damper system data.
 17. The method ofclaim 1, wherein a communication to the wearable device enablingwarnings from the wearable device when the user's hand motions exceedthe set of the parameters for the acceptable motions.
 18. The method ofclaim 1, further comprising: updating the model of the person's bodymotion with updated motion data.
 19. A system for detecting spatialpositioning between a user's hands and an object, using a wearabledevice, to provide feedback to a user, which comprises: a computersystem comprising; a computer processor, a computer-readable storagemedium, and program instructions stored on the computer-readable storagemedium being executable by the processor, to cause the computer systemto perform the following functions to; receive, at a computer, motiondata of a person's body motion while performing an action, the motiondata including sensor data from sensors at a location, the sensorsdetecting the person's body motion, and the sensors including a wearabledevice on the person's body; model, using the computer, the person'sbody motion using the motion data; determine, using the computer, a setof parameters for acceptable motions based on an action risk assessmentof the action; and initiate feedback, using the wearable device, to theperson based on the person's body motion exceeding a body motionthreshold based on the set of parameters for the acceptable motions andthe model.
 20. A computer program product for detecting spatialpositioning between a user's hands and an object, using a wearabledevice, to provide feedback to a user, the computer program productcomprising a computer readable storage medium having programinstructions embodied therewith, the program instructions executable bya computer to cause the computer to perform functions, by the computer,comprising the functions to: receive, at a computer, motion data of aperson's body motion while performing an action, the motion dataincluding sensor data from sensors at a location, the sensors detectingthe person's body motion, and the sensors including a wearable device onthe person's body; model, using the computer, the person's body motionusing the motion data; determine, using the computer, a set ofparameters for acceptable motions based on an action risk assessment ofthe action; and initiate feedback, using the wearable device, to theperson based on the person's body motion exceeding a body motionthreshold based on the set of parameters for the acceptable motions andthe model.